Skip to content

[core] Support snapshot-based sequence ordering for primary-key tables#7832

Open
JunRuiLee wants to merge 7 commits into
apache:masterfrom
JunRuiLee:snapshot-ordering-v2
Open

[core] Support snapshot-based sequence ordering for primary-key tables#7832
JunRuiLee wants to merge 7 commits into
apache:masterfrom
JunRuiLee:snapshot-ordering-v2

Conversation

@JunRuiLee
Copy link
Copy Markdown
Contributor

Purpose

close #7806

Tests

  • SchemaValidationTest#testSnapshotSequenceOrderingHappyPath
  • SchemaValidationTest#testSnapshotSequenceOrderingRejectsSequenceField
  • SchemaValidationTest#testSnapshotSequenceOrderingRejectsNonPkTable
  • KeyValueWithLevelNoReusingSerializerSnapshotIdTest#testRoundTripWithSnapshotId
  • KeyValueWithLevelNoReusingSerializerSnapshotIdTest#testRoundTripWithoutSnapshotId
  • SortMergeSnapshotOrderingTest#testLaterSnapshotWinsOverHigherSequence
  • SortMergeSnapshotOrderingTest#testFallsBackToSequenceWhenSnapshotMissing
  • SortMergeSnapshotOrderingTest#testSameSnapshotFallsBackToSequence
  • SortMergeSnapshotOrderingTest#testStampedAlwaysBeatsUnstamped
  • PrimaryKeySimpleTableTest#testSnapshotSequenceOrdering
  • PrimaryKeySimpleTableTest#testSnapshotSequenceOrderingFallsBackToSequenceWithinSnapshot
  • PrimaryKeySimpleTableTest#testSnapshotSequenceOrderingCompactionPreservesInputSnapshotId
  • PrimaryKeySimpleTableTest#testSnapshotSequenceOrderingWithChangelogInput
  • PrimaryKeySimpleTableTest#testSnapshotSequenceOrderingWithChangelogLookup
  • PrimaryKeySimpleTableTest#testSnapshotSequenceOrderingDeleteFromLaterSnapshot

@JunRuiLee JunRuiLee force-pushed the snapshot-ordering-v2 branch from 36b0eaf to 2c737da Compare May 13, 2026 02:35
@JunRuiLee
Copy link
Copy Markdown
Contributor Author

Hi @JingsongLi, could you help take a look? Many thanks.


public KeyValueDataFileRecordReader(
FileRecordReader<InternalRow> reader, RowType keyType, RowType valueType, int level) {
this(reader, keyType, valueType, level, KeyValue.UNKNOWN_SNAPSHOT_ID);
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remote this constructor

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

stamp = 0;
for (ManifestEntry entry : files) {
if (entry.kind() == FileKind.DELETE) {
stamp = Math.max(stamp, entry.file().minSequenceNumber());
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you double check here?
If there is a compact between file a (snapshot = 1) and file c (snapshot = 3). Then we get file d (snapshot = 3). But the file b (snapshot = 2) does not participate in the compaction. So file b remains snapshot = 2. File b and File a have key range overlapping. So the may merge.

Before compaction, file a sequence number(snapshot id) is smaller than file b.
After compaction, file a (which now is file c) sequence number is larger than file b?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

stamp = 0;
for (ManifestEntry entry : files) {
if (entry.kind() == FileKind.DELETE) {
stamp = Math.max(stamp, entry.file().minSequenceNumber());
Copy link
Copy Markdown
Contributor

@leaves12138 leaves12138 May 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It too dangerous here, after compaction file may have a sequence number far exceed the real snapshot it they write.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed

@JunRuiLee
Copy link
Copy Markdown
Contributor Author

Thanks @leaves12138 for the review! Fixed the compaction ordering issue by persisting per-record snapshotId through _SEQUENCE_NUMBER column. Added tests for the scenario you described. Old constructor removed.

PTAL, Thanks!

@JunRuiLee JunRuiLee force-pushed the snapshot-ordering-v2 branch from 09ba5c9 to 37cc344 Compare May 14, 2026 07:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] Support snapshot-based sequence ordering for primary-key tables

2 participants